<html>
<head><meta charset="utf-8"><title>Memory Profiling · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html">Memory Profiling</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="167114406"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167114406" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167114406">(Jun 02 2019 at 09:38)</a>:</h4>
<p>One of the major problems with rust-analyzer is high memory consumption (unlike rustc, rust-analyzer needs to work with the whole workspace).</p>
<p>I wonder if we can improve visibility into <em>what</em> takes memory?</p>
<p>Currently, analyzer status shows the total heap size (using jemalloc).</p>
<p>I think breakdown by the <em>type</em> of heap object in memory would be extremely useful. However, I don't know <em>how</em> can we get those stats?</p>
<p>Does anyone have an idea?</p>



<a name="167114994"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167114994" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167114994">(Jun 02 2019 at 09:52)</a>:</h4>
<p>I've been intending to try out <a href="https://github.com/nokia/memory-profiler" target="_blank" title="https://github.com/nokia/memory-profiler">https://github.com/nokia/memory-profiler</a> with RA</p>



<a name="167115067"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167115067" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167115067">(Jun 02 2019 at 09:54)</a>:</h4>
<p>Looks interesting! I've tried using heaptrack, but it wasn't too useful. I think both heaptrack and memory-profiler analyze allocations by call-stack, and what I really want, I thinks, is by object type.</p>



<a name="167115076"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167115076" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167115076">(Jun 02 2019 at 09:55)</a>:</h4>
<p>Ideally, this should also work in the analysis-status page...</p>



<a name="167115088"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167115088" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167115088">(Jun 02 2019 at 09:55)</a>:</h4>
<p>as for actually tracking counts of objects... just a random idea, but maybe it'd be possible to have a zero-sized <code>AllocationTracker</code> struct that increments a counter on creation/clone and decrements it on drop? which you'd then put into types you want to track <span aria-label="thinking" class="emoji emoji-1f914" role="img" title="thinking">:thinking:</span></p>



<a name="167115136"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167115136" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167115136">(Jun 02 2019 at 09:56)</a>:</h4>
<p>the counter would have to be global by tracked type somehow</p>



<a name="167115140"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167115140" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167115140">(Jun 02 2019 at 09:56)</a>:</h4>
<p>Yeah, might be a good tool to have!</p>



<a name="167115156"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167115156" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167115156">(Jun 02 2019 at 09:57)</a>:</h4>
<p>I'll also try serveo::heap_size_of. This seems like the right approach overall, but I am not sure if it works in practice</p>



<a name="167115419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167115419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167115419">(Jun 02 2019 at 10:04)</a>:</h4>
<p>Perhaps we should just bite the bullet and add <code>#[derive(DeepSizeOf)]</code> to each type in ra <a href="https://docs.rs/deepsize/0.1.1/deepsize/index.html" target="_blank" title="https://docs.rs/deepsize/0.1.1/deepsize/index.html">https://docs.rs/deepsize/0.1.1/deepsize/index.html</a> ?</p>



<a name="167116110"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167116110" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167116110">(Jun 02 2019 at 10:25)</a>:</h4>
<p>it's worth a try</p>



<a name="167126510"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167126510" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167126510">(Jun 02 2019 at 15:27)</a>:</h4>
<p>I think I've found semi-effective strategy to figure out where the memory is</p>
<ul>
<li>disable auto gc</li>
<li>collect ALL queries in GC</li>
<li>use binary search to find the offending query</li>
</ul>



<a name="167126512"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167126512" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167126512">(Jun 02 2019 at 15:27)</a>:</h4>
<p><a href="/user_uploads/4715/MMMjciGdjY8ELNepMphfYHGv/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/MMMjciGdjY8ELNepMphfYHGv/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/MMMjciGdjY8ELNepMphfYHGv/pasted_image.png"></a></div>



<a name="167127010"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167127010" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167127010">(Jun 02 2019 at 15:42)</a>:</h4>
<p>So, looks like macro expansion is by far the largest contributor to the current memory usage of rust-analyzer</p>



<a name="167127013"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167127013" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167127013">(Jun 02 2019 at 15:42)</a>:</h4>
<div class="codehilite"><pre><span></span>        self.query(hir::db::MacroDefQuery).sweep(sweep);
        self.query(hir::db::MacroArgQuery).sweep(sweep);
        self.query(hir::db::MacroExpandQuery).sweep(sweep);
        self.query(hir::db::ParseMacroQuery).sweep(sweep);
</pre></div>


<p>These four queries consume about 1GB for me, from about total 1.5 GB allocated</p>



<a name="167127149"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167127149" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167127149">(Jun 02 2019 at 15:47)</a>:</h4>
<p>cc <span class="user-mention" data-user-id="216201">@Edwin Cheng</span> I think the biggest contributor here is the syntax trees, and I am working on collecting them better.</p>
<p>However, other queries occupy quite a bit of space due to <code>tt::Subtree</code> as well. I wonder</p>
<ul>
<li>if we have some bug in macro expansion that produces expanded code much bigger that it should be (we have that limit on maximum size of expanded code, could it be the case that we hit it sometimes?)</li>
<li>I wonder if, for storing macro args in salsa, we should use more compact representation than a real tree of tokens? Perhaps we should allocate the three into a flat array, or allocate the nodes in the arena?</li>
</ul>



<a name="167127337"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167127337" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167127337">(Jun 02 2019 at 15:52)</a>:</h4>
<p>First, maybe it is because the current <code>TokenBuffer</code> struct is not optimized, it is basically double up the memory usage but it should be temporarily and not affect the salsa storage. (We still use <code>TokenTree</code> to store it)</p>



<a name="167127451"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167127451" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167127451">(Jun 02 2019 at 15:56)</a>:</h4>
<blockquote>
<p>if we have some bug in macro expansion that produces expanded code much bigger that it should be (we have that limit on maximum size of expanded code, could it be the case that we hit it sometimes?</p>
</blockquote>
<p>You can enable <code>RUST_LOG=ra_hir=WARN</code> to see if there is any warning there but i am doubt it is the case.</p>



<a name="167130195"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130195" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130195">(Jun 02 2019 at 17:16)</a>:</h4>
<blockquote>
<p>I wonder if, for storing macro args in salsa, we should use more compact representation than a real tree of tokens? Perhaps we should allocate the three into a flat array, or allocate the nodes in the arena?</p>
</blockquote>
<p>Should we intern <code>Ident</code> and <code>Literal</code> token in general ?</p>



<a name="167130269"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130269" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130269">(Jun 02 2019 at 17:17)</a>:</h4>
<p>I don't think so: they store <code>SmolStr</code> inside, and it doesn't allocate for short strings</p>



<a name="167130405"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130405" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130405">(Jun 02 2019 at 17:21)</a>:</h4>
<p>But we are using Vec&lt;TokenTree&gt; to store the leaf of Subtree. so it should be still allocated, right ?? And I can imagine there a tons of duplication tokens which are keywords and identifiers in token tree.</p>



<a name="167130476"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130476" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130476">(Jun 02 2019 at 17:23)</a>:</h4>
<p>Hm, I think, for something like <code>struct Foo;</code>, all three tokens would be in a single <code>Vec</code>?</p>



<a name="167130481"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130481" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130481">(Jun 02 2019 at 17:23)</a>:</h4>
<p>That is, that <code>Vec</code> allocation is shared among all leaves, so it's not <em>that</em> bad</p>



<a name="167130487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130487">(Jun 02 2019 at 17:23)</a>:</h4>
<p>but getting rid of that Vec would definitely be sweet!</p>



<a name="167130536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130536">(Jun 02 2019 at 17:24)</a>:</h4>
<p>Do you want to try a <code>smallVec</code> version first ?</p>



<a name="167130545"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130545" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130545">(Jun 02 2019 at 17:24)</a>:</h4>
<p>Hm, yeah, smallvec could help here.</p>



<a name="167130558"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130558" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130558">(Jun 02 2019 at 17:25)</a>:</h4>
<p>It would be interesting to get statistics  about how long are subtrees ingeneral</p>



<a name="167130559"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130559" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130559">(Jun 02 2019 at 17:25)</a>:</h4>
<p>but, long term, arena-based solution seems best to me</p>



<a name="167130560"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130560" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130560">(Jun 02 2019 at 17:25)</a>:</h4>
<p>Or maybe not :-)</p>



<a name="167130606"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130606" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130606">(Jun 02 2019 at 17:26)</a>:</h4>
<p>I imagine, for macro-by-example situations where you can paste a part of input stream into the output stream, you could take advantage of structural sharing</p>



<a name="167130608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130608">(Jun 02 2019 at 17:26)</a>:</h4>
<p>in this case, Art-tree based structure would be more benefitial</p>



<a name="167130622"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130622" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130622">(Jun 02 2019 at 17:27)</a>:</h4>
<p>It'll also work better for recursive macros, which process a bit of <code>$($tt:tt)*</code> at a time</p>



<a name="167130757"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130757" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130757">(Jun 02 2019 at 17:32)</a>:</h4>
<p>Thats true. And if i understand correctly , rustc libprocmacro token handling is handle based: <br>
<a href="https://github.com/rust-lang/rust/blob/dc6db14e1cd60012f25be4fd8d2eb96ea5b4bb68/src/libproc_macro/bridge/handle.rs" target="_blank" title="https://github.com/rust-lang/rust/blob/dc6db14e1cd60012f25be4fd8d2eb96ea5b4bb68/src/libproc_macro/bridge/handle.rs">https://github.com/rust-lang/rust/blob/dc6db14e1cd60012f25be4fd8d2eb96ea5b4bb68/src/libproc_macro/bridge/handle.rs</a></p>



<a name="167130802"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167130802" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167130802">(Jun 02 2019 at 17:33)</a>:</h4>
<p>a <code>Group</code>  is defined as struct Group(Handle)</p>



<a name="167131217"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167131217" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167131217">(Jun 02 2019 at 17:45)</a>:</h4>
<p>I just tried to use <code>smallvec</code> instead, but it fail in our case because I forget it is recursive defined :)</p>



<a name="167172423"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167172423" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167172423">(Jun 03 2019 at 09:46)</a>:</h4>
<blockquote>
<p>I'll also try serveo::heap_size_of. This seems like the right approach overall, but I am not sure if it works in practice</p>
</blockquote>
<p>I think it can, work, it's the same approach that Firefox uses: <a href="https://blog.mozilla.org/nnethercote/2015/06/03/measuring-data-structure-sizes-firefox-c-vs-servo-rust/" target="_blank" title="https://blog.mozilla.org/nnethercote/2015/06/03/measuring-data-structure-sizes-firefox-c-vs-servo-rust/">https://blog.mozilla.org/nnethercote/2015/06/03/measuring-data-structure-sizes-firefox-c-vs-servo-rust/</a></p>



<a name="167172439"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167172439" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167172439">(Jun 03 2019 at 09:46)</a>:</h4>
<p>(there should be some older blog posts about that, too)</p>



<a name="167210157"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167210157" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167210157">(Jun 03 2019 at 17:03)</a>:</h4>
<p><span class="user-mention" data-user-id="203546">@Laurențiu Nicola</span> thanks for that link! That makes me think that we need to ping <span class="user-mention" data-user-id="120989">@nnethercote</span>  :)</p>
<p><span class="user-mention" data-user-id="120989">@nnethercote</span> we'd love implement something like <code>about:memory</code> for rust-analyzer. Unlike <code>rustc</code>, <code>rust-analyzer</code> is a long-lived process, so we need to get <em>current</em> memory usage of various structs, and not just "after-the-process exited" usage. </p>
<p>I am somewhat hesitant to use the <code>heapsize</code> crate:</p>
<ul>
<li>it doesn't seem to be too actively maintained</li>
<li>documentation is very sparse. In particular, it's unclear how to tweak <code>derive(HeapSizeOf) </code></li>
<li>in rust-analyzer, a lot of things are behind <code>Arc</code>s, so it's important to support them (I like deepsize's approach to this: <a href="https://docs.rs/deepsize/0.1.1/deepsize/struct.Context.html" target="_blank" title="https://docs.rs/deepsize/0.1.1/deepsize/struct.Context.html">https://docs.rs/deepsize/0.1.1/deepsize/struct.Context.html</a>)</li>
<li><code>heapsize</code> is jemalloc specific, but ideally there should be an approximate mode that works with any allocator</li>
</ul>
<p>Do you think <code>heapsize</code> is a safe bet, long-term?</p>



<a name="167210192"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167210192" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167210192">(Jun 03 2019 at 17:03)</a>:</h4>
<p>(re-ping <span class="user-mention" data-user-id="120989">@nnethercote</span> b/c matklad can't zullip)</p>



<a name="167210614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167210614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167210614">(Jun 03 2019 at 17:09)</a>:</h4>
<p>Ah, I <em>think</em> <a href="https://github.com/servo/servo/tree/master/components/malloc_size_of" target="_blank" title="https://github.com/servo/servo/tree/master/components/malloc_size_of">https://github.com/servo/servo/tree/master/components/malloc_size_of</a> is what I probably want, but it's in servo :-(</p>



<a name="167210751"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167210751" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167210751">(Jun 03 2019 at 17:10)</a>:</h4>
<p>I suppose we'd have to instrument <code>salsa</code> as well?</p>



<a name="167210790"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167210790" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167210790">(Jun 03 2019 at 17:11)</a>:</h4>
<p><span class="user-mention" data-user-id="203546">@Laurențiu Nicola</span> ideally, yes, but I think just measuring keys/values themselves would be good enough, and that we can do without touching salsa</p>



<a name="167242399"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167242399" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167242399">(Jun 03 2019 at 23:36)</a>:</h4>
<p>@matklad: I advise against using heapsize. I implemented it in Servo. It was kind of experimental. Some eager beaver (can't remember who) put it on <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a>. It has some major problems, such as not handling Arcs.</p>



<a name="167242542"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167242542" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167242542">(Jun 03 2019 at 23:39)</a>:</h4>
<p>malloc_size_of is much better, and fixes most of the problems with heapsize. However, it relies on some mozjemalloc-specific features, such as the ability to get the size of an allocation when you have an interior pointer (i.e. a pointer that points somewhere in the allocation, but not necessarily at the start). I would also still class it as somewhat experimental.</p>



<a name="167242564"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167242564" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167242564">(Jun 03 2019 at 23:39)</a>:</h4>
<p>These comments are worth reading: <a href="https://github.com/servo/servo/blob/master/components/malloc_size_of/lib.rs#L11-L47" target="_blank" title="https://github.com/servo/servo/blob/master/components/malloc_size_of/lib.rs#L11-L47">https://github.com/servo/servo/blob/master/components/malloc_size_of/lib.rs#L11-L47</a> and <a href="https://github.com/servo/servo/blob/master/components/malloc_size_of/lib.rs#L102-L105" target="_blank" title="https://github.com/servo/servo/blob/master/components/malloc_size_of/lib.rs#L102-L105">https://github.com/servo/servo/blob/master/components/malloc_size_of/lib.rs#L102-L105</a></p>



<a name="167242643"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167242643" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167242643">(Jun 03 2019 at 23:40)</a>:</h4>
<p>@Matklad: looks like Webrender has a cut-down fork of malloc_size_of <a href="https://github.com/servo/webrender/blob/master/wr_malloc_size_of/lib.rs" target="_blank" title="https://github.com/servo/webrender/blob/master/wr_malloc_size_of/lib.rs">https://github.com/servo/webrender/blob/master/wr_malloc_size_of/lib.rs</a></p>



<a name="167242740"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167242740" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167242740">(Jun 03 2019 at 23:42)</a>:</h4>
<p>One awkward thing about malloc_size_of is where it sits in the module hierarchy. Ideally it would be a built-in rust thing, and then data structures like HashSet could be built on top of it, rather than it being built on top of HashSet, if you see what I mean.</p>



<a name="167242936"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167242936" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167242936">(Jun 03 2019 at 23:46)</a>:</h4>
<p>In general, live memory measurements are useful, and that's why somebody put heapsize on <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a>. But it's also something of a tricky problem given current language constraints. malloc_size_of is good enough for Servo, but not a truly general-purpose solution that's appropriate for putting on <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a>, IMO.</p>



<a name="167243018"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167243018" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167243018">(Jun 03 2019 at 23:48)</a>:</h4>
<p>Thanks! I've also found another fork of malloc_size_of on <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a>: <a href="https://crates.io/crates/graphannis-malloc_size_of" target="_blank" title="https://crates.io/crates/graphannis-malloc_size_of">https://crates.io/crates/graphannis-malloc_size_of</a></p>



<a name="167249187"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167249187" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> njn <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167249187">(Jun 04 2019 at 01:56)</a>:</h4>
<p>Interesting. Maybe graphannis-malloc_size_of might be a good place to start</p>



<a name="167265844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167265844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167265844">(Jun 04 2019 at 07:36)</a>:</h4>
<p>Yeah, I guess, I would either use that, or publish <code>ra_malloc_size_of</code> to <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a> :-) Thanks again, this all was <strong>super</strong> useful!</p>



<a name="167272883"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167272883" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167272883">(Jun 04 2019 at 09:08)</a>:</h4>
<p>What's the issue with <code>deepsize</code>?</p>



<a name="167274218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167274218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167274218">(Jun 04 2019 at 09:27)</a>:</h4>
<p>at the moment, it doesn't support enums</p>



<a name="167274240"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167274240" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167274240">(Jun 04 2019 at 09:27)</a>:</h4>
<p>enums?</p>



<a name="167274306"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167274306" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167274306">(Jun 04 2019 at 09:28)</a>:</h4>
<p>yeah, it just panics on <code>SmolStr</code>, because that contains an <code>enum</code> inside</p>



<a name="167274331"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167274331" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167274331">(Jun 04 2019 at 09:29)</a>:</h4>
<p>is that a fundamental limitation, or is just not implemented?</p>



<a name="167274345"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167274345" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167274345">(Jun 04 2019 at 09:29)</a>:</h4>
<p>just not implemented</p>



<a name="167274364"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167274364" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167274364">(Jun 04 2019 at 09:29)</a>:</h4>
<p>Like, all of {deepsize,heapsizeof,malloc_size_of} work approximately the same and are not complex in theory</p>



<a name="167274382"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167274382" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167274382">(Jun 04 2019 at 09:30)</a>:</h4>
<p>The question is, which one is closest to being production ready and safe to use across the crate graph</p>



<a name="167501544"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167501544" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167501544">(Jun 06 2019 at 17:18)</a>:</h4>
<p>I managed to get some stats for <code>subtree</code> count by inserting <code>dbg(subtree.count)</code> in macro expanding code:</p>



<a name="167501551"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167501551" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167501551">(Jun 06 2019 at 17:18)</a>:</h4>
<div class="codehilite"><pre><span></span>total = 32234
mean = 122.89470745175902
median = 64.0
max = 34222
The first 100 subtree.count :

count |  occurred
58    =&gt; 2286
46    =&gt; 1535
69    =&gt; 1063
24    =&gt; 868
81    =&gt; 867
0     =&gt; 741
74    =&gt; 573
87    =&gt; 564
23    =&gt; 473
53    =&gt; 413
25    =&gt; 406
18    =&gt; 401
93    =&gt; 400
218   =&gt; 398
57    =&gt; 390
99    =&gt; 376
7     =&gt; 344
21    =&gt; 342
101   =&gt; 328
54    =&gt; 312
32    =&gt; 296
34    =&gt; 287
29    =&gt; 279
105   =&gt; 276
27    =&gt; 273
43    =&gt; 265
28    =&gt; 262
48    =&gt; 256
70    =&gt; 256
15    =&gt; 255
84    =&gt; 252
51    =&gt; 246
22    =&gt; 242
35    =&gt; 232
44    =&gt; 227
39    =&gt; 225
31    =&gt; 224
72    =&gt; 223
111   =&gt; 220
64    =&gt; 219
33    =&gt; 218
63    =&gt; 217
90    =&gt; 208
37    =&gt; 203
50    =&gt; 192
47    =&gt; 192
41    =&gt; 191
26    =&gt; 190
61    =&gt; 183
76    =&gt; 181
117   =&gt; 180
103   =&gt; 176
96    =&gt; 165
91    =&gt; 162
77    =&gt; 152
52    =&gt; 150
17    =&gt; 148
30    =&gt; 145
60    =&gt; 143
79    =&gt; 141
193   =&gt; 135
38    =&gt; 132
302   =&gt; 127
73    =&gt; 126
56    =&gt; 124
55    =&gt; 123
59    =&gt; 122
12    =&gt; 120
42    =&gt; 118
129   =&gt; 117
49    =&gt; 117
45    =&gt; 117
108   =&gt; 115
106   =&gt; 113
115   =&gt; 112
40    =&gt; 108
135   =&gt; 107
16    =&gt; 107
126   =&gt; 104
114   =&gt; 104
102   =&gt; 102
67    =&gt; 102
97    =&gt; 99
88    =&gt; 99
83    =&gt; 96
112   =&gt; 95
123   =&gt; 94
78    =&gt; 94
75    =&gt; 92
94    =&gt; 88
95    =&gt; 84
65    =&gt; 84
62    =&gt; 82
66    =&gt; 82
141   =&gt; 80
147   =&gt; 79
113   =&gt; 78
10    =&gt; 78
80    =&gt; 78
207   =&gt; 74
</pre></div>



<a name="167501623"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167501623" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167501623">(Jun 06 2019 at 17:19)</a>:</h4>
<p>By running <code>ra-cli anaylsis-stats</code> in ra-analyizer</p>



<a name="167502091"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167502091" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167502091">(Jun 06 2019 at 17:25)</a>:</h4>
<blockquote>
<p>64</p>
</blockquote>
<p>That looks like SmallVec wouldn't help much?</p>



<a name="167502251"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167502251" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167502251">(Jun 06 2019 at 17:27)</a>:</h4>
<p>I think so.</p>



<a name="167502261"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167502261" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167502261">(Jun 06 2019 at 17:27)</a>:</h4>
<p>How about <code>TreeArc</code> ?</p>



<a name="167502412"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167502412" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167502412">(Jun 06 2019 at 17:29)</a>:</h4>
<p>Hm, <code>TreeArc</code> works only with syntax trees, it's not a general-purpose smart pointer. Some kind of arena-based solution seems best here</p>



<a name="167502565"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167502565" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167502565">(Jun 06 2019 at 17:31)</a>:</h4>
<p>And forget to write it down : the sum of all count is 3961388</p>



<a name="167502705"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167502705" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167502705">(Jun 06 2019 at 17:32)</a>:</h4>
<p>And note that the <code>dbg</code> is inserted in <code>crate::ids::macro_expand_query</code> , so it is basically the number of tokentree stored in salsa</p>



<a name="167561072"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167561072" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167561072">(Jun 07 2019 at 09:19)</a>:</h4>
<p>probably a stupid question, but could we compress some of the data?</p>



<a name="167561157"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167561157" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167561157">(Jun 07 2019 at 09:20)</a>:</h4>
<p>I assume most space is taken by salsa values (not keys), and those are trees more often than not?</p>



<a name="167561697"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167561697" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167561697">(Jun 07 2019 at 09:29)</a>:</h4>
<p>Yeah, most of the space is occupied by the parse trees</p>



<a name="167561735"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/167561735" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#167561735">(Jun 07 2019 at 09:30)</a>:</h4>
<p>the recent PR adds an LRU for them, but I wonder if there's a more precise strategy</p>



<a name="169336910"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/169336910" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#169336910">(Jun 30 2019 at 11:50)</a>:</h4>
<p><a href="https://github.com/rust-analyzer/rust-analyzer/pull/1463" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/pull/1463">https://github.com/rust-analyzer/rust-analyzer/pull/1463</a> add <code>--memory-usage</code> flag to <code>analysis-stats</code>:</p>
<div class="codehilite"><pre><span></span>Analysis: 18.062757431s, 891mb allocated 1263mb resident
   206mb MacroExpandQuery
    96mb ParseQuery
    82mb MacroArgQuery
    70mb CrateDefMapQuery
    21mb RawItemsWithSourceMapQuery
     7mb MacroDefQuery
     7mb AstIdMapQuery
     4mb ImplsInModuleWithSourceMapQuery
  3285kb ImplDatumQuery
  2947kb GenericParamsQuery
  2348kb InferQuery
  1901kb BodyHirQuery
  1890kb ParseMacroQuery
  1797kb ImplsForTraitQuery
  1613kb BodyWithSourceMapQuery
   651kb FnDataQuery
   601kb ExprScopesQuery
   359kb ImplsInCrateQuery
   244kb GenericPredicatesQuery
   241kb TypeForDefQuery
   175kb TypeAliasDataQuery
   174kb GenericDefaultsQuery
   153kb CallableItemSignatureQuery
   144kb StructDatumQuery
   124kb EnumDataQuery
    46kb StructDataQuery
    38kb TraitDatumQuery
    20kb TraitItemsIndexQuery
    19kb ModuleLangItemsQuery
    17kb TypeForFieldQuery
    15kb LangItemsQuery
    11kb TraitDataQuery
     8kb NormalizeQuery
   4096b ImplementsQuery
      0b SourceRootCratesQuery
      0b RawItemsQuery
      0b ImplsInModuleQuery
      0b ConstDataQuery
      0b StaticDataQuery
      0b LangItemQuery
      0b DocumentationQuery
      0b AssociatedTyDataQuery
</pre></div>



<a name="169338883"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/169338883" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Florian Diebold <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#169338883">(Jun 30 2019 at 12:50)</a>:</h4>
<p>the CrateDefMaps are pretty big, maybe interning the names would help there?</p>



<a name="169340122"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Memory%20Profiling/near/169340122" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Memory.20Profiling.html#169340122">(Jun 30 2019 at 13:30)</a>:</h4>
<p>Yeah, I guess we can just <code>intern</code> all <code>Name</code>s via salsa interning and see how it goes</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>